A Fast Analytical Algorithm for MDPs with Continuous State Spaces

Many real-world domains require that agents plan their future actions despite uncertainty, and that such plans deal with continuous space states, i.e. states with continuous values. While finitehorizon continuous state MDPs enable agents to address such domains, finding an optimal policy is computationally expensive. Although previous work provided approximation techniques to reduce the computational burden (particularly in the convolution process for finding optimal policies), computational costs and error incurred remain high. In contrast, we propose a new method, CPH, to solve continuous state MDPs for both finite and infinite horizons. CPH provides a fast analytical solution to the convolution process and assumes that continuous state values change according to Phase-type distributions. This assumption allows our method to approximate arbitrary probability density functions of continuous state transitions. Our experiments show that CPH achieves significant speedups compared to the Lazy Approximation algorithm, which is the leading algorithm for solving continuous state MDPs.